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Imaging device 
Field 

[0001] The invention relates to an imaging device. Especially the in- 
vention relates to digital imaging devices comprising more than one image 
capturing apparatus. 

Background 

[0002] The popularity of photography is continuously increasing. 
This applies especially to digital photography as the supply of inexpensive digi- 
tal cameras has improved. Also the integrated cameras in mobile phones have 
contributed to the increase in the popularity of photography. 

[0003] For a large group of users, the size of a camera is an impor- 
tant criteria when selecting a camera. A small size camera is easier to carry 
than a large heavy camera. Also the quality of images is naturally important for 
every photographer. However, in traditional lens arrangements the physical 
size of the area (in digital cameras the sensor area) where the lenses produce 
the image determines the focal length and thus the height of the optics, assum- 
ing that the field of view is fixed. Thus, reducing height of the optics and the 
thickness of the camera also reduces the sensor area, and thus the image 
quality. One solution is to reduce the pixel size of the sensor area. Thus the 
resolution and quality is increased. Another solution to reduce theoptics height 
is to divide the sensor area into subsensors which have lens arrangements of 
their own. For example, if the sensor area is divided into four subareas of 
equal size, then each subsensor size is reduced by two. Correspondingly, the 
focal length and thus the optical height are also reduced by two. One known 
solution is presented in US 2002/0089596. 

[0004] Users usually want versatile cameras with several different 
features. Including many features in small scale cameras has so far been diffi- 
cult, if not impossible. 

Brief description of invention 

[0005] An object of the invention is to provide an imaging device 
with versatile features. 

[0006] According to an embodiment of the invention, there is pro- 
vided an imaging device comprising at least one image capturing subsystem of 



a first type, comprising a lens arrangement and configured to produce images, 
at least one image capturing subsystem of a second type comprising a lens 
arrangement, having optical or light gathering properties different from the sub- 
system of first type, configured to produce an image, and a controller config- 
ured to select the subsystem with which an image is to be taken. 

[0007] According to an embodiment of the invention, there is pro- 
vided an imaging device comprising a lenslet array with four lenses, at least 
one sensor array, an image capturing subsystem of a first type comprising a 
lens arrangement, configured to produce images, the lens arrangement of the 
image capturing subsystem of a first type device comprising at least three 
lenses from the lenslet array; and a sensor array, an image capturing subsys- 
tem of a second type comprising a lens arrangement, having optical or light 
gathering properties different from the subsystem of first type, configured to 
produce an image, the lens arrangement of the image capturing subsystem of 
a second type device comprising at least one lens from the lenslet array, and a 
sensor array, and a controller configured to select the subsystem with which an 
image is to be taken. 

[0008] The method and system of the invention provide several ad- 
vantages. In general, the device comprises at least one image capturing sub- 
system of a second type having different light capturing or optical properties 
compared to subsystem of a first type. Thus the user of the device can easily 
take images of different properties without time consuming adjustments or 
without needing to change lens systems. 

[0009] In an embodiment the imaging device of the invention thus 
comprises two image capturing subsystems having different types of optical or 
light gathering properties. 

[0010] In an embodiment, where the invention is applied in a cam- 
era having a lenslet array of lenses, the additional features of the second type 
subsystem are available at a very low cost. For example, the imaging device 
may comprise a lenslet array with at least four lenses and a sensor array. The 
subsystem of the first type may comprise three lenses of the lenslet, each lens 
having a unique color filter from a group of RGB or CMY filters. Thus a color 
image can be produced with the three lenses. The fourth lens may be specially 
designed for different purposes. 

[0011] In an embodiment of the invention, the subsystem of a sec- 
ond type has different optical properties. The subsystem may be, for example, 
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a tele or a macro lens system. Thus the user may take different images of the 
same object very fast. This increases the usability of the camera. For example, 
in low-end cameras it has been necessary, so far, to design the lens arrange- 
ment for general use, as cost and size factors have limited the possibilities of 
"several lenses. With the presented solution even low cost cameras may have 
sophisticated properties, such as lens arrangements suitable for macro or tele 
photography. 

List of drawings 

[0012] In the following, the Invention will be described in greater de- 
tail with reference to the preferred embodiments and the accompanying draw- 
ings, in which 

[0013] Figure 1 illustrates an example of an imaging device of an 
embodiment; 

[0014] Figure 2A and 2B illustrate an example of an image sensing 
arrangement, 

[0015] Figure 2C illustrates an example of color image combining, 
[0016] Figure 3 illustrates an embodiment of the invention; 

[0017] Figure 4 illustrates an embodiment with a macro lens, 
[0018] Figure 5 illustrates an embodiment with a high magnification 

lens, 

[0019] Figure 6 illustrates an embodiment with a tele lens, 

[0020] Figure 7 illustrates an embodiment with a wide-angle lens 

and 

[0021] Figure 8 illustrates an embodiment with an anamorphic lens. 

Description of embodiments 

[0022] Figure 1 illustrates a generalized digital image device which 
may be utilized in some embodiments of the invention. It should be noted that 
embodiments of the invention may also be utilized in other kinds of digital cam- 
eras than the apparatus of Figure 1, which is just an example of a possible 
structure. 

[0023] The apparatus of Figure 1 comprises an image sensing ar- 
rangement 100. The image sensing arrangement comprises a lens assembly 
and an image sensor. The structure of the arrangement 100 will be discussed 
in more detail later. The image sensing arrangement captures an image and 



converts the captured image into an electrical form. The electric signal pro- 
duced by the apparatus 100 is led to an A/D converter 102 which converts the 
analogue signal into a digital form. From the converter the digitized signal is 
taken to a signal processor 104. The image data is processed in the signal 
processor to create an image file. The output signal of the image sensing ar- 
rangement 100 contains raw image data which needs post processing, such as 
white balancing and color processing. The signal processor is also responsible 
for giving exposure control commands 106 to image sensing arrangement 100. 

[0024] The apparatus may further comprise an image memory 108 
where the signal processor may store finished images, a work memory 1 10 for 
data and program storage, a display 112 and a user interface 114, which typi- 
cally comprises a keyboard or corresponding means for the user to give input 
to the apparatus. 

[0025] Figure 2A illustrates an example of image sensing arrange- 
ment 100. The image sensing arrangement comprises in this example a lens 
assembly 200 which comprises a lenslet array with four lenses. The arrange- 
ment further comprises an image sensor 202, an aperture plate 204, a color 
filter arrangement 206 and an infra-red filter 208. 

[0026] Figure 2B illustrates the structure of the image sensing ar- 
rangement from another point of view. In this example the lens assembly 200 
comprises four separate lenses 210 - 216 in a lenslet array. Correspondingly, 
the aperture plate 204 comprises a fixed aperture 218 - 224 for each lens. The 
aperture plate controls the amount of light that is passed to the lens. It should 
be noted that the structure of the aperture plate is not relevant to the embodi- 
ments, i.e. the aperture value of each lens needs not be the same. The num- 
ber of lenses is not limited to four, either. 

[0027] The color filter arrangement 206 of the image sensing ar- 
rangement comprises in this example three color filters, i.e. red 226, green 228 
and blue 230 in front of lenses 210 - 214, respectively. The sensor array 202 
is in this example divided into four sections 234 to 239. Thus, the image sens- 
ing arrangement comprises in this example four image capturing apparatus 
240 - 246. Thus, the image capturing apparatus 240 comprises the color filter 
226, the aperture 218, the lens 210 and the section 234 of the sensor array. 
Respectively, the image capturing apparatus 242 comprises the color filter 
228, the aperture 220, the lens 212 and the section 236 of the sensor array 
and the image capturing apparatus 244 comprises the color filter 230, the ap- 
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erture 222, the lens 214 and the section 238 of the sensor array. The fourth 
image capturing apparatus 246 comprises the aperture 224, the lens 216 and 
a section 239 of the sensor array. Thus, the fourth apparatus 246 does not in 
this example comprise a color filter. 

[0028] The image sensing arrangement of Figures 2A and 2B is 
thus able to form four separate images on the image sensor 202. The image 
sensor 202 is typically, but not necessarily, a single solid-state sensor, such as 
a CCD (Charged Coupled Device) or CMOS (Complementary Metal-oxide 
Semiconductor) sensor known to one skilled in the art. In an embodiment, the 
image sensor 202 may be divided between lenses, as described above, the 
image sensor 202 may also comprise four different sensors, one for each lens. 
The image sensor 202 converts light into an electric current. This electric ana- 
logue signal is converted in the image capturing apparatus into a digital form 
by the A/D converter 102, as illustrated in Figure 1. The sensor 202 comprises 
a given number of pixels. The number of pixels in the sensor determines the 
resolution of the sensor. Each pixel produces an electric signal in response to 
light. The number of pixels in the sensor of* an imaging apparatus es a design 
parameter. Typically in low cost imaging apparatus the number of pixels may 
be 640x480 along the long and short sides of the sensor. A sensor of this reso- 
lution is often called a VGA sensor. In general, the higher the number of pixels 
in a sensor, the more detailed image can be produced.by the sensor. 

[0029] The image sensor 202 is thus sensitive to light and produces 
an electric signal when exposed to light. However, the sensor is not able to 
differentiate different colors from each other. Thus, the sensor as such pro- 
duces only black and white images. A number of solutions are proposed to 
enable a digital imaging apparatus to produce color images. It is well known for 
one skilled in the art that a full color image can be produced using only three 
basic colors in the image capturing phase. One generally used combination of 
three suitable colors is red, green and blue RGB. Another widely used combi- 
nation is cyan, magenta and yellow (CMY). Also other combinations are possi- 
ble. Although all colors can be synthesized using three colors, also other solu- 
tions are available, such as RGBE, where emerald is used as the fourth color. 

[0030] One solution used in single lens digital image capturing ap- 
paratus is to provide a color filter array in front of the image sensor, the filter 
consisting of a three-color pattern of RGB or CMY colors. Such a solution is 
often called a Bayer matrix. When using an RGB Bayer matrix filter, each pixel 
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is typically covered by a filter of a single color in such a way that in horizontal 
direction every other pixel is covered with a green filter and every other pixel is 
covered by a red filter on every other line and by a blue filter on every other 
line. A single color filter passes through to the sensor pixel under the filter light 
which wavelength corresponds to the wavelength of the single color. The sig- 
nal processor interpolates the image signal received from the sensor in such a 
way that all pixels receive a color value for all three colors. Thus a color image 
can be produced. 

[0031] In the multiple lens embodiment of Figure 2A a different ap- 
proach is used in producing a color image. The image sensing arrangement 
comprises a color filter arrangement 206 in front of the lens assembly 200. In 
practice the filter arrangement may be located also in a different part of the 
arrangement, for example between the lenses and the sensor. In an embodi- 
ment the color filter arrangement 206 comprises three filters, one of each of 
the three RGB colors, each filter being in front of a lens. Alternatively also CMY 
colors or other color spaces, may be used as well. In the example of Figure 2B 
the lens 210 is associated with a red filter, the lens 212 with a green filter and 
the lens 214 with a blue filter. Thus one lens 216 has no color filter. As illus- 
trated in Figure 2A, the lens assembly may in an embodiment comprise an in- 
fra-red filter 208 associated with the lenses. The infra-red filter does not nec- 
essarily cover all lenses at it may also be situated elsewhere, for example be- 
tween the lenses and the sensor. 

[0032] Each lens of the lens assembly 200 thus produces a sepa- 
rate image to the sensor 202. The sensor is divided between the lenses in 
such a way that the images produced by the lenses do not overlap. The area 
of the sensor divided to the lenses may be equal, or the areas may be of dif- 
ferent sizes, depending on the embodiment. Let in this example assume that 
the sensor 202 is a VGA imaging sensor and that the sections 234 - 239 allo- 
cated for each lens are of Quarter VGA (QVGA) resolution (320x240). 

[0033] As described above, the electric signal produced by the sen- 
sor 202 is digitized and taken to the signal processor 104. The signal proces- 
sor processes the signals from the sensor in such a way that three separate 
subimages from the signals of lenses 210-214 are produced, one filtered with 
a single color. The signal processor further processes the subimages and com- 
bines a VGA resolution image from the subimages. Figure 2C illustrates one 
possible embodiment to combine the final image from the subimages. This ex- 
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ample assumes that each lens of the lenslet comprises a color filter, in such a 
way that there are two green filters, one blue and one. red. Figure 2C shows 
the top left corner of the combined image 250, and four subimages, a green 
one 252, a red one 254, a blue one 256 and a green one 258. Each of the 
subimages thus comprises a 320x240 pixel array. The top left pixels of the 
subimages correspond to each other and differ only in that the color filter used 
in producing the pixel information is different. Thei subimages are first regis- 
tered. Registering means that any two image points are identified as corre- 
sponding to the same physical point. The top left pixel R1C1 of the combined 
image is taken from the greenl image 252, The pixel R1C2 is taken from the 
red image 254, the pixel R2C1 is taken from the blue image 256 and the pixel 
R2C2 is taken from the green2 image 258. This process is repeated for all pix- 
els in the combined image 250. After this the combined image pixels are fused 
together so that each pixel has all three RGB colors. The final image corre- 
sponds in total resolution with the image produced with a single lens system 
with a VGA sensor array and a corresponding Bayer color matrix. 

[0034] In an embodiment, when composing the final image, the sig- 
nal processor 104 may take into account the parallax error arising from the 
distances of the lenses 210 - 214 from each other. 

[0035] The electric signal produced by the sensor 202 is digitized 
and taken to the signal processor 104. The signal processor processes the 
signals from the sensor in such a way that three separate subimages from the 
signals of lenses 210 - 214 are produced, one being filtered with a single 
color. The signal processor further processes the subimages and combines a 
VGA resolution image from the subimages. Each of the subimages thus com- 
prise a 320x240 pixel array. The top left pixels of the subimages correspond to 
each other and differ only in that the color filter used in producing the pixel in- 
formation is different. Due to the parallax error the same pixels of the subi- 
mages do not necessarily correspond to each other. The parallax error is com- 
pensated by an algorithm. The final image formation may be described as 
comprising many steps: first the three subimages are registered (also called 
matching). Registering means that any two image points are identified as cor- 
responding to the same physical point).Then, the subimages are interpolated 
and the interpolated subimages are fused to an RGB-color image. Interpolation 
and fusion may also be in another order. The final image corresponds in total 
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resolution with the image produced with a single lens system with a VGA sen- 
sor array and a corresponding Bayer color matrix. 

[0036] Figure 3 illustrates an embodiment of the invention. Figure 3 
shows the lens assembly 200, the image sensor 202, the aperture plate 204, 
the color filter arrangement 206 and four image capturing apparatus 240 - 246 
in a more compact form. In this embodiment the image capturing subsystem 
300 of a first type comprises a lens arrangement consisting of the three image 
capturing apparatus 240 - 244. The image capturing subsystem 302 of a sec- 
ond type comprises a lens arrangement consisting of the fourth image captur- 
ing apparatus 246. 

[0037] In an embodiment, the image capturing subsystem 300 pro- 
duces a color image with the three image capturing apparatus 240 - 244. The 
image capturing subsystem 302 comprising the fourth image capturing appara- 
tus 246 may have different light gathering or optical properties compared with 
the other subsystem. The fourth image capturing apparatus 246 may be 
equipped with a Bayer matrix color filter, in which case it is able to produce 
color images. Otherwise it may produce black and white (B/W) or grayscale 
images. 

[0038] In an embodiment, the image capturing subsystem 302 of a 
second type comprises a macro lens arrangement. Macro arrangement means 
that a lens is focused or designed to produce a sharp image from objects near 
the lens. Focusing is usually done by moving the whole lens and/or the image 
sensor to a distance from each other along the optical axis. In a lenslet system 
the fourth lens can be a macro lens and still moulded into the same plastic lens 
block as the other three lenses. For example, if the image capturing subsystem 
300 of a first type has a focus range from 30 cm to infinity, the image capturing 
subsystem 302 of a second type with a macro lens arrangement could have a 
focus range from 10 cm to 30 cm. Thus the total focus range and the usage 
scope of the device are large. 

[0039] In an embodiment, a macro lens is designed to focus near 
and optionally to have field of view different from the other lenses. Thus the 
optical properties of the macro lens are different compared to the other lenses. 
In an embodiment a macro lens uses the same surface form but the lens dis- 
tance from the sensor is longer than in other lenses. Figure 4 illustrates this 
embodiment. Figure 4 shows two lenses of the lens arrangement 200, a nor- 
mal lens 212 used in the subsystem 300 of a first type and lens 216 used as a 
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macro lens in the subsystem 302 of a second type. In addition, the image sen- 
sor 202 is shown. In this embodiment the macro lens 212 is shifted in relation 
to the image sensor. For example, assume that the lens 216 has a fixed focus 
to 500 mm. A focus change from 500 mm to 50 mm for the macro lens requires 
that the lens 216 be shifted 0.76 mm from the image sensor 202, assuming 
that the focal length of the lens is 2 mm. 

[0040] In an embodiment, the image capturing subsystem 302 of a 
second type comprises a high magnification lens arrangement. Thus, the de- 
vice can be used as an electrical magnifier. This can be achieved utilising the 
macro lens arrangement described above in addition to a different lens design. 
In an embodiment the lens is a tele lens, which has macro properties. This has 
the advantage of being able to focus on objects near the lens and to obtain a 
high magnification. For a tele lens the focal length is longer and the field of 
view narrower than in other lenslet camera lenses. 

[0041] Figure 5 illustrates this embodiment. Figure 5 shows two 
lenses of the lens arrangement 200, the normal lens 212 used in the subsys- 
tem 300 of a first type and the lens 216 used as a magnifying lens in the sub- 
system 302 of a second type. In addition, the image sensor 202 is shown. The 
pixel size and resolution are equal in both subsystems. Figure 5 illustrates an 
example where the lens 212 has a focal length of 2 mm and is focused to 500 
mm. The magnifying lens 216 has a focal length of 2.7 mm and is focused to 
50 mm. 

[0042] In an embodiment, the image capturing subsystem 302 of a 
second type comprises a tele lens arrangement. The field of view is smaller for 
a tele lens compared to other lenses. The dimensions of a tele lens are differ- 
ent compared to other lenses. The focal length and the total length of a tele 
lens are bigger. The bigger size can be compensated by designing a smaller 
image sensor area with smaller pixel size for the tele subsystem. When the 
physical dimensions of the image sensor are smaller, also the lens associated 
with the sensor may be smaller. By reducing the pixel size in relation to the 
smaller dimensions the resolution of the tele subsystem remains unchanged. 

[0043] Figure 6 illustrates this embodiment. Figure 6 shows two 
lenses of the lens arrangement 200, the normal lens 212 used in the subsys- 
tem 300 of a first type and the lens 216 used as a tele lens in the subsystem 
302 of a second type. In addition, the image sensor 202 is shown. The pixel 
size and resolution are equal in both subsystems. Figure 6 illustrates an ex- 
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ample where the lens 212 has a focal length of 2 mm and is focused to 1000 
mm. The tele lens 216 has a focal length of 4.0 mm and it is also focused to 
1000 mm. In the illustrated example the tele lens 216 has a double magnifica- 
tion compared, to other lenses. 

[0044] In an embodiment, the image capturing subsystem 302 of a 
second type comprises a wide-angle lens arrangement. The field of view is 
larger for a tele lens compared to other lenses. The dimensions of a wide- 
angle lens are different compared to other lenses. In a wide-angle lens the fo- 
cal length is smaller than in normal lenses. 

[0045] Figure 7 illustrates this embodiment. Figure 7 shows two 
lenses of the lens arrangement 200, the normal lens 212 used in the subsys- 
tem 300 of a first type and the lens 216 used as a wide-angle lens in the sub- 
system 302 of a second type. In addition, the image sensor 202 is shown. The 
pixel size and resolution are equal in both subsystems. Figure 7 illustrates an 
example where the lens 212 has a focal length of 2 mm and is focused to 500 
mm. The wide-angle lens 216 has a focal length of 1 .5 mm and it is focused to 
500 mm. 

[0046] In an embodiment, the image capturing subsystem 302 of a 
second type comprises an anamorphically cylindrical lens. An anamorphic lens 
is designed for a different field of view in vertical and horizontal directions. With 
such a lens it is possible to take very wide-angle images, for example pano- 
rama images. In an embodiment the image capturing subsystem 302 com- 
prises a lens that has a wide horizontal view. The lens of the subsystem pro- 
duces on the image sensor an image, which is compressed in horizontal direc- 
tion. The processor producing the final image corrects distortions arising from 
the anamorphic lens in a manner known for one skilled in the art. 

[0047] Figure 8 illustrates this embodiment. Figure 8 shows two 
jenses of the lens arrangement 200, a normal lens 212 used in the subsystem 
300 of a first type and in the subsystem 302 of a second type an anamorphic 
lens 800 in front of a lens 216 of a normal design. The anamorphic lens modi- 
fies the image in the above described manner so that panorama images, for 
example, can be taken. In addition, the image sensor 202 is shown. In the em- 
bodiment of Figure 8, the lens 800 has a horizontal field of view of 48° and a 
vertical field of view of 40°. The pixel size and resolution are equal in both sub- 
systems. In an embodiment the lens 216 itself is designed to be anamorphic, in 
which case there is no need for lens 800. 
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[0048] Referring to Figures 1 and 2B, in an embodiment the image 
capturing subsystem 302 of a second type comprises a Bayer matrix color filter 
232 or a corresponding color matrix filter and thus it is able to produce color 
images. The processor 104 is configured to take images with the subsystems 
300 and 302 in sequence with small intervals so that fast motion objects can 
be captured on images. The exposure times of the subsystem may be syn- 
chronized such that the image taken by the subsystem 302 is captured with a 
programmable delay from the subsystem 300. The delay can be very short, as 
electronic shutters are used. 

[0049] In an embodiment, the second subsystem is arranged to take 
images in the same color space as the first subsystem. In this embodiment 
both subsystems may produce grayscale images, for example. 

[0050] In an embodiment, the resolution of the image produced by 
the image capturing subsystem of a second type is some resolution commonly 
used in applications such as videoconferencing, where the required resolution 
can be much smaller than in still imaging. Examples of such formats are CIF 
(Common Intermediate Format) and QCIF (Quarter CIF). CIF comprises 288 
lines and 352 pixels per line. QCIF comprises one fourth of the size of CIF, that 
is 144 lines and 176 pixels per line. This may be achieved, for example, by 
selecting the number of pixels in the image sensor of the subsystem 302 ac- 
cordingly. Thus, the image taken by the subsystem 302 may be directly used in 
videoconferencing applications without any conversions that may be needed to 
change the image into the required format This reduces delay and the current 
consumption in a portable device. 

[0051] Even though the invention is described above with reference 
to an example according to the accompanying drawings, it is clear that the in- 
vention is not restricted thereto but it can be modified in several ways within 
the scope of the appended claims. 



